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Kilchan Choi, Michael Seltzer, Joan Herman, and Kyo Yamashiro 
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Abstract 

The No Child Left Behind Act (NCLB, 2002) establishes ambitious goals for increasing 
student learning and attaining equity in the distribution of student performance. Schools 
must assure that all students, including all significant subgroups, show adequate yearly 
progress toward the goal of 100% proficiency by the year 2014. In this paper, we wish to 
illustrate an alternative way of measuring AYP that both emphasizes individual student 
growth over time, and focuses on the distribution of student growth between performance 
subgroups. We do so through analyses of a longitudinal dataset from an urban school district 
in Washington. We also examine what these patterns tell us about schools that meet their 
AYP targets and those that do not. This alternative way of measuring AYP helps bring to 
light potentially important aspects of school performance that might be masked if we limit 
our focus to classifying schools based only on current AYP criteria. In particular, we are able 
to identify some schools meeting the Washington state criteria for AYP, for example, that 
have above average students making substantial progress but below average students 
making little to no progress. In contrast, other schools making AYP have below average 
students making adequate progress but above average students making little gains. These 
contrasts raise questions about the meaning of "adequate" progress - and to whom the 
notion of progress is referring. We believe that closely examining the distribution of student 
progress may provide an important supplementary or alternative measure of AYP. 



Introduction 

The No Child Left Behind Act (NCLB, 2002) establishes ambitious goals for 
increasing student learning and attaining equity in the distribution of student 
performance. Schools must show they are making "adequate yearly progress," or AYP, 
toward the goal of all students achieving proficiency in mathematics and 
reading /language arts by the year 2014. In addition, schools must assure that all 
subgroups of students make such progress, including students who are economically 
disadvantaged, students with disabilities, and students with limited English 
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proficiency. As such, the NCLB not only requires AYP, but also an equitable 
distribution of student learning. 

Attending to the relationship between where students start (i.e., initial status) and 
how rapidly they progress (i.e., their rates of change) can provide us with an indicator 
of whether such equity goals are being achieved and whether particular subgroups are 
being left behind. In other words, examining the relationship between initial status and 
rates of change provides insight into how equitably student achievement is distributed 
within a school (Seltzer, Choi & Thum, 2002, 2003). If a school's average growth rate is 
high, and the relationship between initial status and growth over time is strongly 
positive, we know that children are being left behind. In this case, lower achieving 
students are making less progress than their higher achieving peers. As a result, initial 
gaps in achievement among students become magnified over time. This inequity in 
growth patterns is masked by the high overall growth in the school. On the other hand, 
if a school's average growth rate is low, and the relationship between initial status and 
growth over time is negative, or even weakly positive, this would be an indication that 
the school is doing a reasonable job of leaving no child behind, despite the low overall 
gains. In this case, progress is being underestimated by the low overall growth rate in 
the school. 

The methodology for measuring growth and the equitable distribution of progress 
proposed in this paper pertains to estimating not only school mean growth, but also the 
expected growth or progress of certain subgroups. Our subgroups are defined by their 
initial performance status rather than by their ethnic, language, disability, or income 
levels. For example, we provide expected growth patterns for students who perform at 
a school's mean initial starting value, as well as for students performing 15 points above 
and below the school's average starting point. 

In our analyses, our criteria for adequate progress is defined by, and driven by, 
individual students' gains over a two-year period. This notion of adequate progress, 
though different from AYP in NCLB, still addresses many of the underlying goals and 
tenets of NCLB. In the state of Washington, for example, the AYP decision is currently 
made based primarily on the performance of fourth graders on the Washington 
Assessment of Student Learning (WASL). AYP in Washington and elsewhere is defined 
as a change in the percent proficient from year to year for different cohorts of students 
as they pass through the same grade level (fourth grade). This is often called a multiple- 
cohort design. Similarly, the definition of AYP that most states currently employ 
focuses on school-level progress — in the form of percent of students meeting 
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proficiency in a particular grade — instead of individual-level progress (Thum, 2003). 
Our notion of adequate growth, by contrast, emphasizes monitoring each individual 
student's progress longitudinally. For example, we measure growth over two 
subsequent years, using students' test scores at Grade 3 in 2001, and at Grade 5 in 2003. 
This is an example of a longitudinal panel data design. In this type of design, it is 
possible to examine how much an individual student progresses each year, or over a 
span of 2 or 3 years, as well as whether there are differences in progress across different 
subgroups. 

We first classified schools based on the AYP criteria used by Washington (see, 
Washington State Department of E ducation website: 
http:/ / www.kl2.wa.us/ESEA/presentations.aspx) in the 2001-2002 school year. ''AYP 
Schools" are those that met Washington state's AYP targets; "Non-AYP Schools" are 
those that did not. We re-evaluated each of the AYP and Non-AYP schools based on our 
alternative framework of measuring progress. Our analysis leads us to see that our 
alternative measure of progress can compensate for some of the drawbacks of the 
current AYP measure. 



Research Questions 

We address the following questions in this study: 

1. Are there schools that meet AYP yet still have children who are not making 
substantial progress? That is, are some schools classified as AYP leaving some 
children behind? 

2. Are there schools that do not meet AYP yet still enable students to make 
substantial progress? 

3. Do AYP schools achieve a more equitable distribution of student growth in 
achievement than Non-AYP schools? In other words, are schools that meet 
AYP enabling students at all ability levels to make substantial progress? 

4. Are there Non-AYP Schools that are reducing the achievement gap and 
essentially acting to leave no child behind? 



Data 

Our study draws on elementary school data from a large, racially diverse, urban 
school district in the state of Washington, where required testing includes the Iowa Test 
of Basic Skills (ITBS) in Grades 3 and 5 and the WASL in Grade 4. While determinations 
of AYP are based on the WASL, the ITBS results provided us with the data needed for 
analyzing student growth. The difference in the tests, between the one used to measure 
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progress in our analysis and that used by AYP criteria, is admittedly a limitation in our 
comparisons, particularly given differences in the tests' respective alignment with state 
standards. However, in order to show growth over time for individuals in elementary 
schools, we relied on the measure that was used in more than one elementary grade, the 
ITBS. In addition, equating the WASL and the ITBS in order to use adjacent grade levels 
was not an option with the data provided. 

We used the ITBS reading scale scores for the longitudinal analysis. These scale 
scores are vertically equated developmental scores and thus are appropriate for 
examining student change over time. The test scores for third graders in 2001, and the 
test scores for these students when they were in the fifth grade, provided the basis of 
estimating gains in achievement between Grades 3 and 5 for each student in our 
sample. The analysis includes only students who had test scores in both Grades 3 and 5. 
In all, there are 2,524 students in this sample nested within 72 schools. The number of 
students in each school ranges from 13 to 87. The average number of students in this 
sample per school is approximately 35. 

Among the sample of 72 schools, 51 are AYP schools (or met the AYP criteria) and 
the remaining 21 are Non- AYP schools (or did not meet the AYP criteria). As mentioned 
earlier, in the state of Washington, decisions regarding AYP are based on the state's 
standards-based test, the WASL. At the elementary school level, the WASL is 
administered only at Grade 4. Decisions whether each elementary school met AYP in 
the baseline year (2001-02) and during the second year (2002-03) were made based on 
fourth grade WASL test scores. In this study, the AYP and Non-AYP grouping is based 
on the baseline year (2001-02), when our sampled students were in the fourth grade. 

Note importantly that we used standard errors of measurement (SEs) connected 
with students' ITBS test scores in our analysis. The SE for various ranges of test scores 
was obtained from the technical manual of the ITBS test booklet. As will be seen in 
detail in the following section, the SE information is incorporated into our analyses in 
order to estimate the true initial status and the true gain score for each student in each 
school, by accounting for measurement error. 

Estimating True Gain Based on Two-Time Points with the Standard Errors of 
Measurement: Latent Variable Regression Hierarchical Model (LVR-HM_Gain) 

We use an advanced two-level hierarchical modeling technique in a fully Bayesian 
framework (Choi & Seltzer, 2003; Seltzer, Choi & Thum, 2003; see also Raudenbush & 
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Bryk, Chpt. 11, 2002 for a maximum likelihood based approach). This modeling 
methodology incorporates latent variable regression techniques into two-level 
hierarchical models. In a Level-1 (within-individual) model, time-series observations 
are modeled as a function of a time metric, and we obtain estimates of initial status and 
rates of change for individual students. In a Level-2 (between-individual) model, rates 
of change for each student are modeled as a function of their initial status. The 
coefficient capturing the relationship between initial status and rates of change 
represents an expected increase (in the case of a positive coefficient) or decrease (in the 
case of a negative coefficient) in rates of change when initial status increases one unit. 

We apply this modeling technique to settings where we have observations for 
individuals at two points in time, and the corresponding standard errors of 
measurement for both observations. In the following Level-1 model, ITBS test scores at 
Grades 3 and 5 for each of N students (z = 1, ..., N) (Y f( ) are modeled as a linear function 
of time ( Time ti ). Time ti takes a value of 0 for Grade 3 and a value of 1 for Grade 5. 



Y ti = n 0l + 7i, , Time ti + e ti e ti ~ N (0, [SE(Y f ,)] 2 ) 
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By virtue of our time metric, % oi represents initial status for student z, and n u 
captures gain or change during the span of time. Note that e ti are residuals assumed to 
be normally distributed with a mean of zero and variance represented by [SE(Y h )] 2 . 

To incorporate the standard errors of measurement (SEs) into our model, we re- 
pose a Level-1 model of the following form: 
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where we scale the left and right hand sides of Equation 1 by the inverse of the SEs for 
student z such that £*,, and £* 2( are both assumed to be normally distributed and have a 
mean of zero, but their variance is now 1 (e* tl ~ N (0, 1). By re-scaling outcomes and the 
time metric based on an estimate of the precision associated with each test score, (1 / 
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SE(Y f ,)), it is possible to estimate true initial status (tc 0 ,) and true gain (tt i; ) for student i 
(see Bryk, Thum, Easton & Luppescu, p. 135, 1998). Note here that both ;t 0 , and jq, are 
latent variables, not observed variables. 

We now pose the following Level-2 (between student) model: 

% = Poo r oi r 0 i ~ N (0, x 00 ) (3a) 

Thi= Pi0+ Hn 0i - Poo)+ hi r u ~ N (0, x n ) Cov(r 0 i/ r v ) = 0 (3b) 

In Equation 3a, [3 00 represents the population mean initial status and r oi a random 
effect capturing deviations in the initial status of students from [3 00 . In Equation 3b, true 
gain for student i is modeled as a function of true initial status. Thus, the latent variable 
regression coefficient b captures the relationship between initial status and amount of 
gain. In other words, b captures the expected increase or decrease in the amount of gain 
(71, ,) when initial status (7i o; ) increases one unit (see for example, Raudenbush & Bryk, 
2002; Seltzer et al., 2003). P 10 now represents the expected gain for student i whose initial 
status value (ti* 0 ,) is equal to the mean initial status (p oo ) for that student's school. As for 
variance components, x 00 captures the extent to which students vary in initial status, and 
x n represents a residual variance in gain after taking into account differences in initial 
status. 

Including initial status as a predictor for the gain helps address an important 
question on how student growth is distributed within a school (Seltzer et al., 2003). The 
questions are as follows: Does a student who starts out with higher scores gain more 
than a student who starts out with lower scores? Is the gap among students diminishing 
or increasing over time? As mentioned above, the latent variable regression coefficient, 
b, gives us a sense of the patterns underlying the relationship between expected gain 
and initial status. By attending this b coefficient, we can address a question on what the 
expected gain would be for students at different levels of initial status. More 
specifically, we examine the expected gain for: 1) a student whose initial status value is 
equal to the school mean initial status, 2) a student who is initially starting 15 points 
below the school mean initial status, 3) a student who is initially starting 15 points 
above the school mean initial status. 
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Estimating the expected values at different levels of initial status based on the LVR- 
HM_Gain 

We fit each school's data to the model described in Equations 2a, 3a, and 3b. 
Note that all analyses of the LVR-HM_Gains presented in this paper are conducted 
using the software program WinBUGS1.4 (i.e., the Windows version of Bayesian 
Analysis Using the Gibbs Sampler developed by Spiegelhalter et al, 2002). WinBUGS 
provides a fairly easy means of implementing the Gibbs sampler in a wide range of 
modeling settings. Upon convergence, the Gibbs sampler essentially provides us with 
an accurate approximation of the marginal posterior distribution for all of the 
parameters in our model. Using the resulting marginal posterior distributions, we can 
obtain the posterior mean, median, standard deviation, and 95% interval based on the 
.025 and .975 quantiles of the marginal posterior distribution. 

Using the resulting estimates after fitting the model depicted in Equations 2a, 3a 
and 3b, we first estimate the expected scores of students in Grades 3 and 5, depending 
upon three sets of student initial status: equal to the school mean, 15 points below the 
school mean, and 15 points above the school mean. Then we estimate the expected gain 
by subtracting the expected value at Grade 3 from the expected value at Grade 5. 

1. Expected gain for students initially starting at school mean initial status 

We pose equations for the mean initial status and the mean gain as follows: 

E Oho_M I t =0, K oi = Poo) = poo (4a) 

E 0%_m I 1 = 1/ = Poo) = Poo+ Pi o (4b) 

The expected value at t = 0 is equal to the school mean initial status (i.e., p oo ), and 
the expected value at t = 1 is equal to the mean initial status plus the mean gain (i.e., p oo 
+ P 10 ), since the second part in the right hand side of Equation 3b involving the b 
coefficient is equal to 0 (i.e., (7t or Poo) = 0, thus b x (jc 0i - - p oo ) = 0). As a result, for a student 
who starts at the school mean initial status, the expected gain is obtained by subtracting 
Equation 4a from Equation 4b, and it is equal to the mean gain (i.e., P 10 ) as in the 
following equation: 

Gainjnean = E (Y tl M ) - E (Y t0M ) = (Poo + Pio) - Poo = Pio (4c) 

2) Expected gain for students initially starting 15 points below the school mean 
initial status 
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The expected values at Grades 3 and 5 for students starting out 15 points below the 
school mean initial status are specified in the following Equations 5a and 5b: 

E (Y t0 L I t =0, 7C 0 , = Poo - 15) = p 00 + (-15) (5a) 

E (Y tl L I t =1, n oi = poo - 15) = (p 00 + (-15)) + (p :o + b x (-15)) (5b) 

Where the expected value at t = 0 can be obtained by subtracting 15 points from 
the school mean initial status ((3 00 ). hi Equation 5b, we estimate the expected value at t 
=1. In this equation, b x (-15) captures the extent to which we expect an increment or 
decrement from the mean gain (P 10 ) when the initial status is 15 points below the mean 
initial status. This quantity is then added to the expected initial status 15 points below 
the mean initial status. If the coefficient b has a positive value, for example, the expected 
gain is smaller than the mean gain. In contrast, if the coefficient b has a negative value, 
the expected gain becomes larger than the mean gain (P 10 ), since the magnitude of ( b x (- 
15)) is added to the mean gain. After estimating expected values at t = 0 and t =1, we 
next estimate the expected gain for students starting out 15 points below the school 
mean in the following equation: 

Gain_15 pts below-avg = E (Y tl L ) - E (Y t0 L ) 

= [(p 00 + (-15)) + (p 10 + b x (-15'))] - (p 00 + _ (-15) = (P 10 + b x (-15)) (5c) 

Where the expected gain is obtained by subtracting Equations 5a from Equation 
5b. Since the first quantity in the right hand side of Equation 5b is equal to the quantity 
in the right hand side of Equation 5a, this value cancels out and we end up with the 
second quantity in Equation 5b. 

Note that when implementing the Gibbs sampler, the quantity of Gain_15 pts 
below-avg is simply accomplished by subtracting the deviates for Y (k) t0 L from Y (k) tl L in 
each iteration of the algorithm (i.e., Y ( V - Y (k) t 0 _ L ). As such, we can obtain the estimate 
as well as the 95% interval based on the .025 and .975 quantiles of the marginal posterior 
distribution of Gain_15 pts below-avg . 

3) Expected gain for students initially starting 15 points above the school mean 
initial status 

Similar to the equations above, for a student whose initial status is 15 points above 
the school mean, we pose the following equations: 



E (Yto_L I t —0, 7i 0( — (3 00 + 15) — (3 00 + 15 
E (Y tl _ L I t =1, 7i o; = p 00 + 15) = (p 00 + 15) + (P 10 + (b x 15)) 



(6a) 

(6b) 




9 



The expected value at t = 0 is equal to the mean initial status plus 15 points. Similar 
to Equation 5b, the expected value at t = 1 is equal to the expected value at t = 0 plus the 
expected gain (i.e. (p 10 + (bx 15)). We next estimate the expected gain for students starting 
out above the school mean in the following equation: 

Gain_15 pts above-avg = E (Y tl H ) - E (Y t0 H ) 

= [((3 00 + (15)) + (P 10 + (b x 15))] - (p 00 + 15) = p 10 + (b x 15) (6c) 

As above, the expected gain is accomplished by subtracting the expected value at t 
— 0 from the expected value at t = 1. 



Results 

Descriptive statistics of data. The district in our sample is a diverse, urban district 
with several schools that do not meet AYP criteria. As mentioned earlier, of the 72 
schools in our sample, 51 are AYP schools and 21 are Non- AYP schools. As can be seen 
in Table 1, the percent of students eligible for free /reduced price lunch in AYP schools 
is 24.7% (c.f. district average = 36.4%). In comparison, 69.6% of students in Non-AYP 
schools are eligible for free /reduced price lunch, which is almost double the district 
average. In terms of racial composition, more than half of the students in AYP schools 
(56.5%) are White. In contrast, African American (38.5%) and Asian students (33.4%) 
together compose the majority of the population in Non-AYP schools. 
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Table 1 

AYP schools vs. Non-AYP Schools on Measures of the Mean ITBS Reading Scores, the Percentage of 
Students Eligible for Free /Reduced Price Lunch, and Ethnic Composition 



AYP school (N = 51) Non AYP school (N = 21) 





# of 

Students 


Mean 


SD 


# of Students 


Mean 


SD 


tTBS reading at 














Grade 3 


1866 


196.2 


21.1 


658 


176.5 


16.4 


Grade 5 


1866 


226.7 


23.9 


658 


203.9 


20.8 


% eligible for 
free / reduced 




24.7% 






69.6 % 




price lunch 














Ethnic 


White: 56.5%, Afr Am: 11 


.9% 


White: 12.9%, Afr Am: 38.5% 




composition 


Hispan: 


9.0%, Nativ Am.: 


3.1% 


Hispan: 12.5%, 


. Nativ Am.: 2.7% 






Asian: 19.7% 




Asian: 33.4% 







As to the outcome variable, the district means of ITBS reading scores at Grades 3 
and 5 are 191.0 and 221.8, respectively, and the corresponding standard deviations are 
21.7 and 25.2. The district average gain between Grades 3 and 5 is approximately 30 
points, which is larger than one pooled standard deviation. 

When looking at the AYP schools and Non-AYP schools separately, AYP schools 
have higher mean scores than Non-AYP schools at Grade 3 by approximately 20 points, 
and their mean gains are slightly larger than the mean gain for Non-AYP schools. On 
this superficial level, judged by the different mean scores, we might at first glance tend 
to agree with the AYP classifications made by the state. However, when looking more 
closely at student growth trajectories of students staring out at the school mean, 15 
points below the school mean, and 15 points above the school mean, we find patterns of 
growth that might call those classifications into question. 

Comparisons between AYP schools and Non-AYP schools. In our final analysis, we 
compare the state's AYP classifications against the results that we obtained by fitting 
each school's data to the expected gains described above (e.g., LVR-HM_Gain presented 
in Equations 4c, 5c, and 6c). Specifically, the comparison is made based on the expected 
gains for each of the three different levels of initial status. First, for each of the 72 
schools, we compare AYP and Non-AYP schools' average expected gains. This 
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comparison provides us with information on a.) whether there are AYP schools that, on 
average, are not making substantial yearly progress against our measures, or b.) whether 
there are Non-AYP schools that, on average, are making substantial yearly progress 
against our measures. Second, we compare AYP schools and Non-AYP schools based 
on the magnitude of the expected gain for students 15 points below the school mean. By 
conducting this comparison focusing on those students starting below the school 
average, we clearly see whether those that need the most help are being left behind in 
schools that have been classified as having met AYP criteria. In addition, we can see 
whether there are exceptional Non-AYP schools that are reducing the achievement gap 
and moving all children forward at a substantial rate. Finally, we also compare AYP 
and Non-AYP schools based on the expected gains for students starting out 15 points 
above the school mean. Looking at growth trajectories of students starting above the 
school mean allows us to see if students that are above average are accelerating at 
different rates than others in the school; that is, if students that started out above 
average are languishing in certain schools. 

1) The expected mean gain (AYP schools vs. Non-AYP schools) 

As can be seen in Table 2, we tabulate expected gains for AYP and Non-AYP 
schools. Note importantly that we classify schools depending upon whether each 
school's 95% interval of expected gain is above or below the district average gain. In 
other words, our classification is based on not only the point estimate of the expected 
mean gain, but also the estimation error around this point estimate, as can be clearly 
seen in Figures la through 3b. In these figures, the horizontal line represents the district 
average gain for our sampled 72 schools, and the bars represent each school's expected 
mean gain and its 95% interval. The top line, middle circle, and bottom line of each bar 
correspond, respectively, to the upper bound, mean, and lower bound (or the .975 
quantile, mean, and the .025 quantile) of the marginal posterior distribution of the 
expected mean gain for a given school. 

Care needs to be taken in interpreting results presented in the figures in this paper. 
Suppose that we want to make a hypothetical test on whether the expected mean gain 
for a school is higher than the district average gain (i.e., Gainjnean > 30). Consider, for 
example, school #19 in Figure la: The 95% interval around this school's expected mean 
gain spans a 21-point range (23 to 44), which overlaps with the district average. This 
means that we fail to reject the original hypothesis under a traditional two-tail test at p 
=.05 that this school's expected mean gain is greater than (or less than) the district 
average of 30. However, given a traditional one-tail test at p — .05 of the hypothesis that 
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the school's expected mean gain is greater than the lower bound of its confidence 
interval, 97.5% of the mass of the posterior distribution is greater than 23 points. Thus, 
we can make a statistical statement with 95% confidence that the expected mean gain is 
greater than 23. As such, the .025 quantile value, or the lower bound of the interval in 
these figures (la, lb, 2a, 2b, 3a, and 3b), is the lower bound value we can use to make a 
statistically significant statement about the expected value for any given school. 



Table 2 

Cross Tabulation of AYP Schools Vs. Non-AYP Schools with Respect to the Amount of Expected Gain at 
Different Levels of Initial Status 





AYP school (N= 51) 


Non-AYP school (N= 21) 


Student Initial 


> 30 a 


<10 a 


> 30 a 


<10 a 


Status Values 










School mean 


12 schools 


1 school 


2 schools 


1 school 




(7,8,9,10,22,25,29, 


(15) 


(23, 55) 


(50) 




32,34,45,68,72) 








15 points 


9 schools 


3 schools 


5 schools 


3 schools 


above the 
school mean 


(7, 8, 9, 25, 28, 32, 


(15, 19, 43) 


(6, 23, 38, 55, 64) 


(17, 27, 50) 




34, 65, 68) 








15 points 


7 schools 


4 schools 


0 schools 


5 schools 


below the 
school mean 


(8, 19, 22, 25, 45, 


(9, 15, 42, 44) 




(37, 38, 40, 48, 50) 




63, 72) 









a n =Lower bound of 95% interval of gain. 

Note. The classification of the schools is made based on the lower bound of 95% interval (i.e., .025 quartile 
of the posterior distribution of the expected gain), not based on the estimate of the expected gain (i.e., 
posterior mean of the expected gain). 



As can be seen in Figure la, 15 schools among 51 AYP schools have a point 
estimate of expected mean gain that is smaller than the district average gain. 
Furthermore, among the rest of the 36 schools, only 12 have a 95% interval situated 
above the district mean gain. In other words, the low end of the 95% interval for these 
12 schools does not cross the district average gain line. Note that we see one AYP school 
(#15) that has an expected mean gain of approximately 13 points, but its 95% interval 
captures a value of 0, even though this school is labeled as "a school meeting 
adequately yearly progress." This school has the smallest sample size (the number of 
students is 13) among our sampled schools, which is reflected in the largest band of 



error. 
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Figure la. Expected mean gain in ITBS reading scores for AYP schools. 

The horizontal line represents the district average gain for our sampled 72 schools. The 
top line, middle circle, and bottom line of each bar correspond, respectively, to the .975 
quantile, mean, and the .025 quantile, of the marginal posterior distribution of the 
expected mean gain for a given school. 



Almost half of the Non-AYP schools (in Figure lb) have point estimates of 
expected gains that are higher than the district average. Among those, we have two 
exceptionally well-performing schools. Schools #23 and #55 have a higher expected 
mean gain than the district mean gain by approximately 5-10 points, and the lower 
bound of their 95% intervals are above the district average gain. Only one Non-AYP 
school has an expected mean gain interval that captures a value of 10. Note that 10 
points is approximately one standard deviation smaller than the district average gain. 
As such, we clearly see that there are remarkably large numbers of Non-AYP schools 
making sizable (and close to district average) gains. 
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school 

Figure lb. Expected mean gain in ITBS reading scores for Non-AYP schools. 
The horizontal line represents the district average gain for our sampled 72 
schools. The top line, middle circle, and bottom line of each bar correspond, 
respectively, to the .975 quantile, mean, and the .025 quantile, of the marginal 
posterior distribution of the expected mean gain for a given school. 



2) The expected gain for students starting out 15 points below the school mean 
(AYP schools vs. Non-AYP schools) 

For each school, we also examine how much gain is expected for students starting 
out 15 points below the school mean. The lack of adequate growth for students that 
start out 15 points below the school mean is a problem for both AYP and Non-AYP 
schools. As can be seen in Figure 2a, approximately half of the AYP schools (i.e., 25 
schools) have point estimates of expected gains that are greater than 30 points. 
However, there are four AYP schools (#s 15, 42, 44, and 9) where their 95% intervals 
capture a value of 10. Among these four schools, two schools' intervals (#s 15 and 9) 
include a value of 0, which strongly indicates that students starting out below the mean 
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in these schools made no gains, and that children who need the most help are being left 
behind in these schools. 
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Figure 2a. Expected gain in ITBS reading scores for a student whose initial status 
value is equal to 15 points below the school mean initial status (AYP school). 
The horizontal line represents the district average gain for our sampled 72 schools. 
The top line, middle circle, and bottom line of each bar correspond, respectively, to 
the .975 quantile, mean, and the .025 quantile, of the marginal posterior distribution 
of the expected gain for a given school. 



Many of the Non- AYP schools in Figure 2b have point estimates (for students 
starting out below the mean) that are slightly below the district average; however, 16 of 
21 Non- AYP schools have intervals that include the district average of 30. We can see 
that some of the Non-AYP schools are doing a decent job for students starting out 
below the school mean. The point estimate of expected gain is larger than the district 
average gain in 5 schools (#s 23, 46, 27, 26, and 18). In addition, for five Non-AYP 
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schools (#s 16, 46, 27, 26, and 18), the expected gain is significantly larger than 20 points. 
However, below average students in five of the Non-AYP schools (#s 50, 48, 40, 38, and 
37) had less than a 10-point gain (see Table 2), indicating that students below the mean 
in these schools are being left behind as well. 




50 48 40 20 38 33 37 24 62 17 64 6 16 13 12 55 23 46 27 26 18 

school 



Figure 2b. Expected gain in ITBS reading scores for a student whose initial status value 
is equal to 15 points below the school mean initial status (Non-AYP school). 
The horizontal line represents the district average gain for our sampled 72 schools. The 
top line, middle circle, and bottom line of each bar correspond, respectively, to the .975 
quantile, mean, and the .025 quantile, of the marginal posterior distribution of the 
expected gain for a given school. 



3) The expected gain for students starting out 15 points above the school mean 
(AYP schools vs. Non-AYP schools) 

Figures 3a and 3b display each school's expected mean gains for students starting 
out 15 points above the school mean. These figures clearly illustrate that a substantial 
proportion of schools — both AYP (69%) and Non-AYP (57%) — have point estimates for 
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the expected gain for their above average students that are higher than the district 
average gain. This implies that students who initially start higher than the school 
average in a given school are very likely to make adequate gain, regardless of whether 
they are in AYP or Non-AYP schools. 




15 19 49 1 63 2 42 58 4 53 67 51 71 39 31 66 54 41 57 56 47 22 34 29 32 7 
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Figure 3a. Expected gain in ITBS reading scores for a student whose initial status value 
is equal to 15 points above the school mean initial status (AYP school). 
The horizontal line represents the district average gain for our sampled 72 schools. The 
top line, middle circle, and bottom line of each bar correspond, respectively, to the .975 
quantile, mean, and the .025 quantile, of the marginal posterior distribution of the 
expected gain for a given school. 
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Figure 3b. Expected gain in ITBS reading scores for a student whose initial status 
value is equal to 15 points above the school mean initial status (Non-AYP 
school). 

The horizontal line represents the district average gain for our sampled 72 
schools. The top line, middle circle, and bottom line of each bar correspond, 
respectively, to the .975 quantile, mean, and the .025 quantile, of the marginal 
posterior distribution of the expected gain for a given school. 



It is noteworthy that there are several AYP and Non-AYP schools in which 
students with high initial status make large gains. In AYP School #7, for example, the 
estimate of expected gain is approximately 51 points and the low end of its 95% interval 
is close to 40 points. Likewise, in three Non-AYP schools (#s 6, 55, and 23), the estimates 
of expected gains for above average students are greater than 40 points and the 
corresponding 95% interval falls above the district average gain. 

A more interesting finding, however, is that above average students in one AYP 
school made no gains. We can identify this type of school by looking at whether the 
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95% interval of the expected gain captures a value of 0 (see school #15). Three AYP 
schools (#s 15, 43, and 19) and three Non- AYP schools (#s 50, 27, and 17) also have 
intervals that include 10 points or less. These findings beg the question of whether 
classifications such as AYP are sufficient if above average students do not experience 
any changes or little gains. 

The distribution of student gain in AYP schools and Non- AYP schools. In the above 
sections, we compared AYP schools with Non-AYP schools in terms of the expected 
mean gain, and the expected gains for below average and above average students. 
These analyses show that examining the expected mean gain alone does not provide a 
complete picture of whether schools may be leaving children behind. Examining the 
expected gains by different levels of initial status helps us understand how progress 
varies by performance levels. In this section, we re-examine the distribution of student 
gains by combining the three performance levels into one picture of school growth. As 
can be seen in Figure 4, we bring the three expected gain trajectories into one picture. 
This figure combines Figures la, 2a, and 3a in the sense that the circle, triangle, and box 
lines, respectively, correspond to expected gains for above average, average, and below 
average students in a given school. Note also that the dotted line (with a diamond icon) 
represents the district average gain. 

We present three different types of AYP schools in Figure 4 (also see Table 3): 

a. Type I: Substantial gain across all performance subgroups, i.e.. No Child 
Feft Behind (School #8). We define this Type I school as having the lower bound of 
the 95% interval around the expected gain for each of the three different 
performance subgroups all greater than the district average gain of 30 points. 

b. Type II: No adequate gain for above average students; substantial gain for 
below average students (school #19). In Type II schools, the lower bound of the 
95% interval around the expected gains for above average students and average 
students is less than 30, while the lower bound for below average students is 
greater than 30. 

c. Type III: No adequate gain for below average students; substantial gain for 
above average students (School #28). In contrast to Type II schools. Type III 
schools are defined as having the lower bound of the 95% interval around the 
expected gain for average and below average students that is less than 30, while 
the lower bound for above average students is greater than 30. 
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Table 3 

Distribution of Student Gain for AYP Schools 





15pts > 


school mean 


Equal to the school mean 


15pts < 


school mean 




Estimate 


95% interval 


Estimate 


95% interval 


Estimate 


95% interval 


Type I 


School #08 


37.1 


(30.7, 43.8) 


38.7 


(35.2, 42.2) 


40.3 


(35.0, 45.7) 


School. 


36.3 


(30.0, 43.2) 


36.9 


(31.7, 42.0) 


37.4 


(31.1, 43.9) 


School. 


37.7 


(30.8, 42.8) 


36.4 


(32.2, 40.5) 


36.0 


(31.1,41.1) 


Type If 


School #19 


22.1 


(6.9, 35.9) 


33.5 


(22.8, 44.1) 


44.8 


(30.7, 60.3) 


School #63 


28.2 


(18.2, 39.0) 


34.1 


(27.8, 40.3) 


40.0 


(30.5, 49.6) 


Type III 


School #28 


41.2 


(33.0, 51.2) 


31.2 


(27.4, 35.1) 


21.2 


(12.1, 28.7) 


School #65 


35.4 


(31.4, 39.5) 


31.3 


(28.5, 34.0) 


27.1 


(23.7, 40.4) 



Note. Expected gains for students whose initial status value is equal to the school mean initial status, at 15 
points above the school mean initial status, and at 15 points below the school mean initial status. 



These categories of school growth allow us to note another trend in the growth 
patterns of these schools, one that involves the closing or widening of achievement gaps 
in these schools. In Type I schools, the growth trajectories for the three performance 
subgroups are parallel, while the other types of schools have growth trajectories that 
either converge or diverge. In Type II schools, the initial gap between above average 
and below average students in the school closes over time, while the initial gap gets 
magnified in Type III schools. Much attention is paid to the closing of the achievement 
gap, and while this phrase is most often used to refer to the racial gap in achievement 
that exists in many schools. Type II schools in our sample are making progress in 
closing the gap between an important pair of subgroups — those that start out below 
average and those that start about above average. When evaluating a school's 
effectiveness or quality, it is difficult from an ethical point of view to choose the better 
of Type II or Type III schools. However, from a policy point of view, more resources 
and sanctions are levied against schools in the Type II category, e.g., through Title I (i.e.. 
No Child Left Behind) funds. 
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First, consider the distribution of student gains for the exemplary AYP schools 
presented in Figure 4. We see the distribution of student gains for School #8 as an 
example of the Type I schools. In this school, it can be clearly seen that all three 
expected gain trajectories are parallel to each other, and steeper than the district mean 
gain trajectory. Note that this school has only 5.7 % of students eligible for free/ reduced 
price lunch. Two other AYP schools (#22 and #25) are classified as Type I schools (see 
Table 3). 

Type II AYP schools (#19 and #63) do not seem to be making substantial progress 
with their above average students, but are succeeding with their below average 
students. In school #19 (featured in Figure 4), the point estimate of the expected gain for 
above average students is 22.1 and the 95% interval ranges from 6.9 to 35.9. In contrast, 
for below average students, the estimate of the expected gain is equal to 44.8 and its 
95% interval ranges from 30.7 to 60.3. Given this large difference between above and 
below average students in their expected gain, the initial gap is substantially 
diminished over time. Closing the gap among students is very important in addressing 
equity concerns; however, this exemplary Type II growth pattern presents problems 
around the lack of progress for above average students. 

Type III AYP schools (#28 and #65) demonstrate a reverse growth pattern of that 
found in Type II schools. As can be seen in Figure 4, the gap in school #28 is magnified 
in this school, since below average students are making far slower progress than above 
average students. In school #28, the estimate of the expected gain for above average 
students is 41.2 (its 95% interval ranges from 33.0 to 51.2), which is approximately 
double the expected gain for below average students (expected gain = 21.2; 95% interval 
= [12.1, 28.7]). While both schools (#28 and #65) met current AYP criteria, these schools 
are not adequately, or equitably, distributing gains among their student subgroups. 
Those that need the most help in these schools seem to be struggling more. Thus, while 
these schools have been identified as making adequate yearly progress, this analysis 
raises the question: Adequate progress for whom? 
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Type II 


Type III 


Gain: high initial = low initial 


Gain: high initial < low initial 
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Figure 4. Distribution of student gain for AYP schools. Type I School: No Child Left Behind 
School with a substantial gain of more than 30 points across all performance subgroups. 

Type If School: No adequate gain (less than 30 points) for students initially starting 15 points 
above the average, while substantial gain (larger than 30 points) for students initially starting 15 
point below the average. Type III School: No adequate gain (less than 30 points) for students 
initially starting 15 points helow the average, while substantial gain (larger than 30 points) for 
students initially starting 15 point above the average. 



Similarly, we present Non- AYP schools with the same three types of distribution 
of student gain in Table 4 and Figure 5. Though the amount of gain is slightly smaller 
than for Type I AYP schools, one Non- AYP school (#26) displays the most equitable 
Type I growth pattern (all subgroups growing in parallel). As we can see in Figure 5, 
school mean initial status for School 26 is approximately 20 points below the district 
average mean initial status, and even the above average subgroup starts out slightly 
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below the district average. However, all three expected trajectories are, by and large, 
parallel to — or slightly steeper than — the district average trajectory. Specifically, the 
estimates of expected gain are 32.8, 32.2, and 31.6, respectively, for above average, 
average, and below average students. Given the higher proportion of low SES students 
in this school (66.0% of students are eligible for free /reduced lunch), the amount of 
expected gain and the distribution of student gains that this school achieves would 
seem to merit special attention, despite the fact that this school did not meet AYP 
criteria. 



Table 4 

Distribution of Student Gain for Non- AYP Schools 





15pts > school mean 


Equal to the school mean 


15pts < school mean 




Estimate 


95% interval 


Estimate 


95% interval 


Estimate 


95% interval 


Type I 


School #26 


32.8 


(24.1,41.7) 


32.2 


(27.6, 36.9) 


31.6 


(22.5, 40.4) 


Type II 


School #27 


18.7 


(9.0, 27.2) 


24.6 


(17.6, 31.5) 


30.5 


(21.5, 40.1) 


Type III 


School #06 


40.7 


(31.5, 50.8) 


32.4 


(26.2, 38.5) 


24.1 


(13.6, 33.7) 


School #38 


39.2 


(30.0, 49.9) 


29.9 


(25.5, 34.2) 


20.5 


(9.3, 30.2) 


School #64 


37.3 


(31.2, 44.2) 


30.6 


(26.9, 34.4) 


24.0 


(17.0, 30.2) 



Note. Expected gains for students whose initial status value is equal to the school mean initial status, at 15 
points above the school mean initial status, and at 15 points below the school mean initial status. 



In addition. School #27 represents a Type II Non-AYP school that appears to be 
struggling. Almost all students in this school are eligible for free /reduced price lunch. 
The mean initial status is 25 points below the district average. However, this school is 
doing a reasonable job of making progress with many students, particularly the below 
average students: the estimate of the expected mean gain is close to 25 points and the 
expected gain for below average students is greater than 30 points. As a result, the gap 
between above and below average students is closing. In fact, the intervals for the 
expected gains of the three performance subgroups are overlapping in Grade 5, 
indicating that the gap remaining between the subgroups is fairly negligible. 
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Finally, School #6 represents our Non-AYP Type III School, where above average 
students are progressing at a faster rate than are others in the school. Again, however, 
this school is another struggling school, as demonstrated by the higher performing 
students in this school starting out below the district average. Those students near the 
district average, however, made the steepest gains (i.e., estimate = 40.7, 95% interval = 
[31.5, 50.8]), with those students starting below the school mean gaining far less (i.e., 
estimate = 24.1, 95% interval = [13.6, 33.7]). 
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Figure 5. Distribution of student gain for AYP schools. 

Type I School: No Child Left Behind School with a substantial gain of more than 30 points 
across all performance subgroups. Type II School: No adequate gain (less than 30 points) for 
students initially starting 15 points above the average, while substantial gain (larger than 30 
points) for students initially starting 15 point below the average. Type III School: No 
adequate gain (less than 30 points) for students initially starting 15 points below the average, 
while substantial gain (larger than 30 points) for students initially starting 15 point above the 
average. 
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Discussion 

Our analyses of estimating the expected gains across different performance 
subgroups provides an informative picture of how student gain is distributed within a 
school. This alternative way of measuring progress helps bring to light potentially 
important aspects of school performance that might be masked if we limit our focus to 
classifying schools based on a single schoolwide measure, or measure that focuses on 
changes at a cut-point. In particular, we are able to identify some AYP schools in which 
above average students hardly make any progress but low-performing students make 
substantial progress. This kind of AYP school may be closing the achievement gap by 
helping their below average students, but they are doing little for their higher 
performing students. One might say the above average students are being left behind. 
In contrast, we also found that in some of the Non- AYP schools, below average students 
are making substantial progress, indicating that certain kinds of progress get 
overlooked by the AYP classifications. 

These contrasts raise questions about the meaning of "adequate" progress, for 
whom we wish to target progress, and the basis on which we label schools as making 
adequate yearly progress or not. We believe that closely examining the distribution of 
student achievement may provide an important supplementary or alternative measure 
of AYP. Our hope is that our approach provides a more informative picture of student 
progress beyond a single AYP indicator and may stimulate discussion among teachers 
and administrators to identify students in need earlier (see Seltzer et al., 2003). We also 
believe that encouraging educators to think about achievement levels rather than (or in 
addition to) current subgroup categories may be both more productive and actionable. 

Our new approach to AYP measures can be extended in many ways. First, our 
approach could easily include differences in student- and school- characteristics. As we 
saw in an exemplary Non- AYP school, and as the literature suggests, low SES schools 
have a more difficult time in making AYP than do high-SES schools and more diverse 
schools have a harder time making AYP than to less diverse schools (Novak & Fuller, 
2003). As we know, it is possible to take these factors into account in evaluating school 
performance. If we re-estimate the distribution of student progress using a model where 
we include differences in student background and in school characteristics (Choi & 
Seltzer, 2003), we might have even more unfavorable results for schools that are 
currently designated as making AYP. 
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It is also possible to extend our approach in settings where we wish to compare the 
performance of different cohorts of students. Our analyses focused on one cohort of 
students who were in Grade 3 in 2001. However, we might see some differences 
between this cohort and the 2000 or the 2002 cohort of students in this district. Using 
multiple measures in multiple cohorts of students (i.e., multi-panel multiple-cohort 
data), we would be able to examine differences in initial status and expected growth 
among and between the cohorts of students (see Bryk, et al, 1998). This longitudinal 
perspective is important because particular school reform efforts or changes in school 
characteristics that may impact achievement schoolwide or at a particular grade level 
might take place in some years and not others (e.g., increment of school budget, 
numbers of qualified teachers, student demographic composition, and so on). This 
multi-panel, multi-cohort design would increase the robustness of our design by adding 
a longitudinal component, and would also help us to track the cohort differences due to 
some of these unmeasured factors. 
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